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FOREWORD 


This  project  was  conducted  in  response  to  a  Chief  of  Naval  Education  and  Training 
(N-5)  request  to  evaluate  the  performance-related  enabling  skills  training  (PREST) 
project,  a  computer-based  approach  to  the  Navy's  academic  remedial  training  (ART) 
program.  The  evaluation  was  conducted  at  the  Recruit  Training  Command  (RTC), 
Orlando,  FL  during  FY80.  Results  are  intended  for  use  by  the  Chief  of  Naval  Education 
and  Training  (N-5)  and  the  RTCs. 

We  gratefully  acknowledge  the  assistance  of  CDR  James  Kiefer,  YN2  Pam  Keskeys, 
and  the  other  members  of  the  Academic  Remedial  Training  Division,  RTC  Orlando; 
Dr.  Ron  Bucknam  of  the  National  Institute  of  Education;  Nancy  Ashcroft  of  the  Orange 
County  (Florida)  Board  of  Public  Instruction;  Joe  Small  and  Dick  Wall  of  Control  Data 
Corporation;  and  Joanne  Stolte,  Shirley  Smith,  and  Keith  Kershner  of  Research  for  Better 
Schools,  Inc. 


JAMES  F.  KELLY,  JR.  JAMES  J.  REGAN 

Commanding  Officer  Technical  Director 
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SUMMARY 


Problem  and  Background 

^-/V  The  Navy  may  be  forced  to  use  larger  numbers  of  marginally  qualified  recruits  in  the 
future,  because  of  manpower  shortages.  If  this  occurs,  academic  remedial  training  (ART) 
will  have  to  be  expanded,  resulting  in  a  greater  number  of  instructors.  Automated 
instruction  is  an  attractive  alternative,  since  its  use  can  reduce  the  number  of  instructors 
needed.  Therefore,  to  determine  the  economic  and  technological  feasibility  of  automated 
instruction  for  the  ART  program,  the  Chief  of  Naval  Education  and  Training  (N-5) 
•contracted  for  the  development  and  test  of  a  computer-based  approach,  hereafter 
referred  to  as  the  Performance-related  Enabling  Skills  Training  (PREST)  Program. 

Objective 

The  objectives  of  this  effort  were  to  compare  the  instructional  effectiveness  of  the 
PREST  configuration  with  that  of  the  standard  classroom  approach,  quantify  the  dif¬ 
ferences  in  effectiveness  in  terms  of  cost  savings,  chart  the  expected  future  costs  of  each 
alternative  as  a  function  of  time,  and  predict  the  point  at  which  the  alternatives  break 
even^ 

Approach 

A  total  of  152  recruits  who  were  enrolled  in  the  ART  program  at  the  Recruit  Training 
Command  at  Orlando  participated  in  the  evaluation.  Of  these,  75,  the  control  group, 
received  ART  through  classroom  instruction;  and  77,  the  experimental  group,  through 
PREST.  Because  of  a  change  in  policy  that  applied  during  2  months  when  only  PREST 
recruits  were  being  tested,  the  overall  entering  reading  grade  level  (RGL)  average  of 
these  recruits  was  one-half  grade  higher  than  that  of  the  classroom  group.  To  compensate 
for  this  difference,  analyses  were  conducted  on  test  scores  obtained  by  restricted 
samples,  which  included  only  those  recruits  with  RGLs  between  5.0  and  6.0.  Also,  for 
both  groups,  restricted  and  overall  samples  were  compared  on  background  measures,  as 
well  as  on  measures  of  immediate  (during  ART)  and  prolonged  (during  the  7-week  recruit 
training  period)  effects  of  instruction  types  on  Navy-related  and  general  reading  skills. 

Findings 

The  two  groups  were  equal  in  instructional  benefits  on  immediate  impact  and  on 
prolonged  benefits.  The  cost  analysis  revealed  that  the  PREST  configuration  was  less 
cost-effective  than  the  classroom  approach  in  1980,  but  that  the  difference  would  steadily 
decline  in  future  years.  The  predicted  point  at  which  the  alternatives  break  even  depends 
on  whether  the  instructors  are  military  or  civilian,  and  on  whether  the  computer  usage 
charge  is  predicted  to  remain  constant  or  decline  five  percent  per  year.  If  civilian 
instructors  are  used  and  the  computer  usage  charge  declines  by  five  percent  per  year, 
PREST  will  become  cost-effective  in  Orlando  in  the  late  1980s. 

Conclusions 


Although  PREST  is  not  now  as  cost-effective  as  the  classroom  approach  to  ART,  it 
nrfciy  become  cost-effective  in  the  future.  Its  cost-effectiveness  can  be  influenced  by 
advances  in  computer  technology  (e.g.,  the  continued  development  of  stand-alone  sys¬ 
tems),  the  type  of  instructor  used  (i.e.,  military  vs.  civilian),  and  the  relative  costs  (e.g., 
salaries,  overhead  costs)  incurred  for  these  instructors. 
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Recommendations 


1.  The  PREST  program  should  not  be  substituted  for  the  current  ART  classroom 
approach  at  this  time. 

2.  Consideration  should  be  given  to  (a)  replacing  the  vocabulary  and  comprehension 
modules  of  ART  with  a  computer-based  system,  and  (b)  using  low-cost,  stand-alone 
microcomputers  as  the  delivery  devices. 

3.  The  worksheets  currently  used  with  PREST  should  be  incorporated  into  the  ART 
study  skills  module. 

4.  The  Navy  Recruit  Reading  Test  should  be  developed  further  as  a  selection  device 
for  ART. 
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INTRODUCTION 


Problem 

The  military  prime  enlistment  pool  (i.e.,  male  high  school  graduates,  aged  18  to  21 
years,  in  the  higher  mental  categories)  will  decrease  significantly  during  the  next  20  years 
(Congressional  Budget  Office,  1977).  Consequently,  the  military  services  may  be  forced 
to  accept  more  recruits  from  the  lower  mental  categories.  Past  experience  has 
demonstrated  that  these  personnel  have  lower  reading  abilities,  are  less  useful  in  the 
service,  and  attrite  more  frequently  than  do  personnel  in  the  higher  mental  categories. 

The  Navy  currently  accepts  small  numbers  of  marginally  qualified  recruits  and  uses 
uniformed  instructors  to  teach  them  basic  skills,  primarily  reading,  as  part  of  the 
academic  remedial  training  (ART)  program  at  the  three  Navy  Recruit  Training  Commands 
(RTCs)  (Orlando,  FL,  Great  Lakes,  IL,  San  Diego,  CA).  The  purpose  of  ART  is  to  improve 
a  recruit's  reading  and  study  skills  so  that  the  recruit  can  successfully  complete  the 
academic  portion  of  recruit  training.  There  is  a  strong  relationship  between  reading 
ability  and  success  in  military  training.  For  example,  Hoiberg,  Hysham,  and  Berry  (1974), 
found  that  the  exit  reading  grade  level  (RGL)  of  ART  graduates  was  the  best  predictor  of 
successful  completion  of  their  first  year  of  Navy  service. 

If  manpower  shortages  force  a  lowering  of  enlistment  standards,  remedial  programs 
like  ART  can  be  expanded  to  accommodate  larger  numbers  of  poorly  qualified  recruits. 
Doing  so,  however,  will  divert  more  instructors  from  their  regular  assignments.  If  the 
Navy  must  accept  increasing  numbers  of  less  qualified  enlistees  and  continue  enrolling 
them  in  the  ART  program,  alternatives  to  the  use  of  military  instructors,  such  as 
civilianization  or  automation,  must  be  considered.  At  present,  civilian  ART  instructors 
are  used,  to  a  limited  extent,  at  no  cost  to  the  Navy  through  local  adult  education 
programs.  These  arrangements,  however,  are  subject  to  change  owing  to  policy  changes 
at  the  local  level.  Thus,  automated  instruction  appears  to  be  an  attractive  alternative  to 
the  standard  classroom  approach,  with  either  civilian  or  military  instructors. 

Background 

ART  Program 

The  current  ART  program  was  standardized  and  implemented  in  October  1978  and 
appears  to  be  successful,  with  over  85  percent  of  its  graduates  successfully  completing 
recruit  training.  The  1979  throughput  was  3266  students,  with  over  10  percent  being 
nonnative  English  speakers. 

The  ART  program  is  based  on  a  diagnostic-prescriptive  approach  to  remedial  reading 
and  is  offered  to  recruits  who  score  between  the  4.0  and  6.0  RGL  on  the  Gates-MacGinitie 
Reading  Test  (1978),  Level  D,  which  is  administered  to  all  Navy  recruits  during  their  first 
week  in  training.  Those  scoring  above  6.0  continue  with  recruit  training  and  those  scoring 
below  6.0  are  given  an  alternate  version  of  the  test  to  confirm  their  initial  low  score.  If 
recruits  score  below  4.0  on  the  second  test,  they  are  referred  to  a  Review  Board.  If  they 
score  between  4.0  and  6.0,  they  are  enrolled  in  ART. 

The  ART  entrants  are  given  the  Stanford  Diagnostic  Reading  Test  (SDRT),  Brown 
Level  (Karlsen,  Madden,  &  Gardner,  1976),  which  diagnoses  weaknesses  in  vocabulary, 
literal  comprehension,  inferential  comprehension,  phonetic  analysis,  and  structural  analy¬ 
sis.  Instructional  modules  in  each  of  these  reading  skill  areas,  which  were  developed  by 
the  Memphis  State  University  under  contract  from  the  Chief  of  Naval  Education  and 


Training  (CNET),  are  then  assigned  to  ART  entrants  according  to  their  diagnosed  needs. 
A  separate  class  is  available  for  each  reading  skill  area,  and  recruits  must  pass  a 
criterion-referenced  module  test  before  they  proceed  to  the  next  class.  A  typical  class 
size  is  about  12  students  per  instructor.  Recruits  require  from  1  to  5  weeks  to  complete 
the  reading  skill  modules,  depending  on  the  number  assigned. 

After  ART  recruits  complete  their  assigned  reading  skills  modules,  they  must 
complete  a  study  skills  module,  which  prepares  them  for  the  academic  curriculum  of 
recruit  training.  (Continued  failure  in  these  academic  tests  can  lead  to  discharge.)  This 
study  skills  module  is  also  required  of  recruits  who  were  not  selected  for  ART  initially, 
but  who  failed  one  or  more  academic  tests  given  as  part  of  recruit  training. 

The  confirmatory  Gates-MacGinitie  and  diagnostic  tests  cure  administered  by  an  ART 
staff  member  and  the  criterion  tests,  by  the  instructors.  All  tests,  from  the  initial  Gates- 
MacGinitie  to  the  criterion-referenced  module  tests,  are  graded  with  an  optical  scanner 
linked  to  the  Navy's  Integrated  Training  and  Resource  Administration  System  in 
Millington,  TN.  Test  administration  and  grading  require  one  full-time  staff  member. 

Performance-related  Enabling  Skills  Training  (PREST)  Program 

With  costs  of  computer  hardware  decreasing  at  a  steady  pace,  prospects  for  less 
expensive,  computer-based  instruction  are  improving  (Wolfe  &  Williams,  1979).  To 
examine  these  prospects,  CNET  contracted  for  the  development  of  a  computer-based  ART 
program  that  focused  on  improving  Navy  functional  reading  and  study  skills.  The  program 
developed  entitled  "performance-related  enabling  skills  training”  (PREST),  called  for  on¬ 
line  reading  and  study  skills  instruction  via  the  Control  Data  Corporation  (CDC)  PLATO 
system,  augmented  by  Navy-related  off-line  drill  and  practice. 

PLATO  is  a  comprehensive,  computer-based  educational  system  that  uses  a  central- 
ized-mainframe  CDC  CYBER  70,  170,  or  6000  series  computer.  From  this  computer, 
instruction  is  downloaded  over  leased  phone  lines  to  the  PLATO  terminal,  a  micropro¬ 
cessor-based  device  with  16K  of  random-access  memory  available  for  storage.  A  read¬ 
only  memory  of  2K  bytes  contains  the  operating  system  and  diagnostic  routines  for 
hardware  maintenance.  The  terminal  provides  high-resolution  computer  graphics,  an 
electronic  keyboard,  and  a  touch-sensitive  panel. 

The  on-line  portion  of  PREST  consists  of  two  introductory  modules,  94  reading 
modules  (selected  from  among  the  111  reading  modules  within  the  Basic  Skills  Learning 
System,  an  adult  basic  skills  curriculum  from  CDC's  existing  courseware  library),  and  six 
specially  developed  study  skills  modules  applicable  to  academic  needs  during  Navy  recruit 
training.  The  computer-based  management  system  of  PREST  routes  each  recruit  through 
on-line  instruction  for  all  of  the  introductory  and  study  skills  modules.  For  on-line 
reading  instruction,  it  prescribes  and  automatically  assigns  modules  based  on  individual 
on-line  diagnosis. 

Off-line  study  skill  application  activities  allow  recruits  to  make  a  transition  from 
learning  via  a  computer  to  learning  via  reading  assignments  and  classroom  lectures,  which 
is  the  learning  mode  used  in  recruit  academic  training.  Supplemental  off-line  reading  drill 
and  practice  materials,  including  those  provided  with  the  Basic  Skills  Learning  system  and 
those  specially  developed  for  PREST,  are  available  to  recruits  who  wish  additional 
practice  or  who  are  waiting  for  terminal  access  to  continue  on-line  reading  activities. 


PREST  was  designed  for  use  by  one  military  instructor  working  with  2k  recruits 
having  access  to  12  PLATO  terminals.  The  instructor's  role  is  to  answer  recruit  questions 
about  the  purpose  of  ART  and  the  operation  of  the  terminals,  clarify  misunderstandings 
about  what  occurs  on-line,  monitor  student  progress,  provide  on-the-spot  instruction  as 
needed,  direct  selected  study  skills  application  activities,  and  provide  feedback  on  some 
off-line  materials. 

The  instructional  objective  of  both  PREST  and  the  standard  classroom  approach  to 
ART  is  to  provide  recruits  with  basic  reading  and  study  skills  needed  to  complete 
successfully  the  academic  portion  of  recruit  training.  Thus,  it  is  important  to  determine 
whether  PREST  is  effective,  in  terms  of  both  instruction  and  cost,  as  compared  to  the 
classroom  approach  to  ART. 

Purpose 

The  purposes  of  the  evaluation  reported  here  were  to  compare  the  instructional 
effectiveness  of  PREST  with  that  of  the  standard  classroom  approach  to  ART,  quantify 
differences  between  the  alternatives  in  effectiveness  in  terms  of  cost  savings,  chart  the 
expected  future  costs  of  each  alternative  as  a  function  of  time,  and  predict  the  point  at 
which  the  alternatives  break  even. 


APPROACH 

Instructional  Effectiveness 

Previous  CAI  cost-effectiveness  evaluations  have  been  criticized  because  they  did 
not  objectively  monitor  students  in  their  follow-on  duty  assignments  (Orlansky  &  String, 
1979).  The  studies  reviewed  by  Orlansky  and  String  relied  on  pre-post  measures  of  student 
performance  during  training,  instructional  time,  academic  attrition,  and  student  and 
instructor  attitudes.  Although  such  measures  are  critical  to  assessing  a  program's 
instructional  effectiveness,  they  do  not  address  the  more  important  operational  issue  of 
job  preparedness.  Also,  postgraduate  effectiveness  measures  often  consist  only  of  a 
supervisor's  opinion,  which  can  be  influenced  by  factors  unrelated  to  the  training  being 
evaluated. 

Because  of  these  criticisms,  the  current  evaluation  was  designed  to  compare  both  the 
immediate  (during  ART)  and  prolonged  (during  the  7-week  recruit  training  period)  effects 
of  classroom  and  PREST  instruction  on  Navy-related  and  general  reading  skills. 


A  total  of  152  recruits  who  were  enrolled  in  the  ART  program  at  RTC  Orlando 
participated  in  the  evaluation.  Of  these,  75,  the  control  group,  received  ART  through 
classroom  instruction;  and  77,  the  experimental  group,  through  PREST.  All  subjects  were 
native  English  speakers. 

Because  the  number  of  recruits  entering  ART  Orlando  in  late  1979  was  less  than 
expected,  55  control  recruits  were  tested  between  October  1979  and  January  1980,  and 
the  other  20,  in  June  and  July,  1980.  The  entire  PREST  group  was  tested  between 
February  and  May,  1980.  In  March  and  April  1980,  when  only  PREST  recruits  were  being 
tested,  a  temporary  change  in  Navy  policy  increased  the  RGL  cutoff  score  for  ART 
selection  from  6.0  to  6.7.  This  change  permitted  21  recruits  with  RGLs  in  that  range  to 
enter  the  PREST  group,  which,  meant,  of  course,  that  the  overall  entering  RGL  average 


of  PREST  recruits  was  higher  than  that  of  the  ciassroom  group.  Further,  the  classroom 
group  included  more  recruits  reading  below  the  5.0  RGL  than  did  the  PREST  group. 
Therefore,  statistical  analyses  were  conducted  on  a  restricted  sample,  which  included  only 
those  recruits  with  RGLs  between  5.0  and  6.0  (47  recruits  for  both  the  classroom  and 
PREST  groups). 

Measures 


The  following  background  data  were  collected  for  each  sample  member:  (1)  the 
scores  obtained  on  the  Armed  Forces  Qualifying  Test  (AQFT)  and  on  the  word  knowledge 
(WK)  subtest  of  the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB),  (2)  entering 
RGL  as  measured  by  the  confirmatory  Gates-MacGinitie  test  (1978),  and  (3)  high  school 
graduation  status. 

To  measure  improvements  in  Navy-related  reading  skills  during  ART,  NAVPER5- 
RANDCEN  developed  two  equivalent  30-item  forms  of  a  Navy  Recruit  Reading  Test 
(NRRT).  The  test  items,  which  were  mainly  multiple-choice  with  an  occasional  sentence 
completion  item,  were  patterned  after  those  in  the  four  recruit  academic  tests  that  are 
administered  during  recruit  training.  For  each  version  of  the  test,  10  questions  were 
designed  to  test  recruits'  ability  to  obtain  information  from  the  table  of  contents  and 
index  of  the  Blue  Jackets  Manual;  and  the  other  20,  to  comprehend  material  in  three 
passages  on  military  subjects. 1  Both  forms  had  been  pilot-tested  at  RTC,  San  Diego. 
Based  on  the  results  of  this  test,  a  time  limit  of  23  minutes  was  judged  as  appropriate. 

The  comprehension  section  of  the  two  1965  versions  of  the  Gates-MacGinitie  Reading 
Test  was  also  used  to  measure  improvement  in  general  reading  comprehension  skills 
during  ART.  Since  ART  students  complete  one  of  the  two  1978  versions  of  this  test 
during  the  first  week  of  training  and  the  other  to  confirm  their  initial  low  score,  using 
them  for  a  posttest  would  have  required  administering  one  form  a  second  time. 

The  final  measure  of  immediate  improvement  was  the  number  of  instructional  days 
recruits  in  the  control  and  experimental  groups  spent  in  ART.  Allowances  were  made  for 
times  when  recruits  were  under  medical  care  or  otherwise  absent  from  the  classroom.  If 
applicable,  reasons  for  discharge  were  recorded. 

The  four  recruit  academic  tests  administered  during  recruit  training  were  used  to 
measure  prolonged  effects  of  the  two  forms  of  ART.  Since  the  expressed  purpose  of  ART 
is  to  enhance  performance  on  the  academic  portion  of  recruit  training,  these  tests  were 
the  best  available  measure  of  ART'S  effectiveness.  The  first  three  tests  each  contained 
50  items;  and  the  final  test,  150  items. 

Design  and  Procedure 

The  design  conformed  to  Campbell  and  Stanley's  "Nonequivalent  Control  Group 
Design”  (1966);  that  is,  recruits  were  assigned  to  one  group  or  the  other,  depending  on  the 
month  when  they  entered  the  Navy.  All  subjects  entering  the  ART  program  were 
administered  one  version  of  the  NRRT  and  Form  1  from  the  1965  Gates-MacGinitie  test. 
When  they  completed  the  program,  they  were  administered  the  alternate  version  of  the 


‘Six  passages  from  the  Blue  Jackets  Manual  or  The  Basic  Military  Requirements 
manual  were  modified  by  shortening  sentences  and  substituting  easy  words  for  difficult 
words.  Three  of  these  passages  were  included  in  one  test  version;  and  the  other  three,  in 
the  other. 


4 


NRRT  and  Form  2  of  the  Gates-MacGinitie  test.  The  alternate  NRRT  forms  were 
counterbalanced  between  pre-and  posttests  within  each  group.  During  recruit  training, 
the  raw  scores  they  obtained  on  the  four  academic  tests  were  recorded.  If  applicable, 
reasons  for  discharge  were  recorded. 

Analyses 


In  both  groups,  restricted  (5.0  to  6.0  RGL)  and  overall  samples  were  compared  on 
background  measures,  as  well  as  on  measures  of  immediate  and  prolonged  effects.  Also, 
for  the  restricted  samples,  analyses  of  variance  (ANOVAs)  were  used  to  compare  pre-and 
posttreatment  NRRT  and  Gates-MacGinitie  test  scores,  as  well  as  the  number  of 
instructional  days  required.  The  scores  obtained  by  the  restricted  samples  on  the  four 
recruit  academic  tests  were  compared  using  a  2  x  4  repeated  measures  ANOVA. 

Cost  Effectiveness 


The  costs  of  the  classroom  and  PREST  approaches  to  ART,  computed  in  constant 
1980  dollars,  were  compared.  For  the  classroom  approach,  it  was  determined  that  four 
instructors  would  be  needed,  based  on  a  1979  throughput  in  Orlando  of  929  recruits.  This 
computation  assumed  an  average  instructor-to-student  ratio  of  12,  12.5  days  of  ART 
instruction  for  the  average  recruit,  250  days  per  year  available  for  instruction,  and  a 
fixed,  month-by-month  flow  of  students  through  ART.  (In  reality,  the  throughput  is  far 
greater  during  the  summer  months.  This  problem  is  addressed  in  the  Discussion  section 
below.)  One  additional  staff  member  would  be  required  to  administer  and  grade  the 
diagnostic  test  required  by  the  classroom  approach,  for  a  total  of  five  military 
instructors.  The  cost  for  these  instructors  was  computed  using  the  billet  costs  for  the 
typical  ART  instructor,  an  E-4  Yeoman  ($14,588)  (see  Koehler,  1980).  The  costs  for  ART 
classroom  materials  (e.g.,  the  Basic  Specific  Skills  Series,  dictionaries,  Skim-and-Scan 
booklets,  vocabulary  bools,  etc.)  were  obtained  from  supply  records.  These  materials 
usually  can  be  recycled  for  use  by  other  recruits;  the  length  of  their  usable  life  varies 
from  1  to  about  4  years,  for  a  cost  of  $2188. 

For  PREST,  costs  were  based  on  the  utilization  of  24  terminals  and  2  instructors, 
since  it  was  determined  that  these  resources  could  handle  a  load  equivalent  to  that 
projected  for  the  classroom  approach.  The  PREST-related  expenses  include  those  for  the 
terminals,  computer  usage  charges,  hardware  maintenance,  and  instructors.  The  costs  for 
the  terminals  were  based  on  a  purchase  plan,  which,  in  the  long  run,  is  far  more  favorable 
than  a  lease  option,  and  were  allocated  over  a  5-year  economic  life,  which  is  considered  a 
reasonable  estimate  of  the  physical/technological  life  of  a  terminal.  The  cost  of 
computer  hardware  has  demonstrated  a  steady  decline,  which  is  predicted  to  continue 
(Noyce,  1977).  According  to  the  cost  figures  supplied  by  CDC,  the  cost  of  PLATO 
terminals  has  been  decreasing  about  10  percent  per  year.  This  decline  was  considered  in 
the  analysis. 


The  costs  for  computer  usage,  which  are  tied  to  system  operator  and  maintenance 
costs  and  profit  motives  are  more  difficult  to  predict.  Two  assumptions  can  be  considered 
here:  either  the  cost  will  remain  constant,  or  it  will  decline  at  5  percent  per  year. 

The  expenditures  that  are  common  to  both  the  classroom  and  PREST  approaches 
(e.g.,  for  classroom  space,  desks,  overhead  lights,  etc.)  were  not  included  in  the  cost 
analysis.  Further,  costs  for  ART  personnel,  such  as  the  division  officer  and  a  yeoman, 
were  not  included  since  these  personnel  would  be  needed  regardless  of  the  instructional 
delivery  used.  A  workbook  by  the  Training  Analysis  and  Evaluation  Group,  entitled 
Improving  Your  Navy  Reading  Skills,  is  used  in  each  approach.  Also,  the  costs  of  the 
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consumable  test-answer  sheets  used  for  the  diagnostic  and  criterion  tests  in  the  classroom 
approach  (estimated  at  $300  per  year)  are  approximately  equal  to  the  costs  for 
reproducing  sufficient  numbers  of  the  consummable,  RBS-developed  worksheets  for 
PREST.  The  costs  of  developing  the  ART  classroom  curriculum  and  the  specially- 
developed  PREST  materials  were  treated  as  "sunk  costs." 

PREST  does  avoid  the  cost  of  the  half  day  required  to  administer  the  diagnostic  test 
and  assign  an  instructional  program.  PREST  does  this  task  on-line  at  the  beginning  of  the 
Basic  Skills  Learning  System  curriculum.  This  amounts  to  a  cost  avoidance  of  $7022, 
which  was  computed  by  multiplying  the  yearly  throughput  by  a  recruit's  half-day  salary. 

The  cost-effectiveness  analysis  was  based  on  a  break-even  analysis,  an  analytical 
technique  used  to  study  the  relationship  between  alternate  cost  patterns  (Fabrycky  & 
Thuesen,  1974).  This  analysis  focuses  on  finding  the  value  of  a  variable,  time  in  this  case, 
at  which  the  economic  desirability  of  two  alternatives  is  equal.  Costs  for  the  classroom 
and  PREST  versions  of  ART  were  projected  over  a  10-year  planning  horizon. 

Since  consideration  is  being  given  to  civilianizing  the  program,  cost  estimates  for 
civilian  instructors  were  obtained  from  the  Orange  County  (Florida)  Board  of  Public 
Instruction,  Adult  Program,  which  currently  supplies  ART  Orlando  with  3.5  civilian 
classroom  instructors  at  no  cost  to  the  Navy.  The  annual  cost  for  civilian  instructors, 
including  overhead,  is  $21,175.  Using  this  figure  to  replace  the  military  costs  and  the 
previously  mentioned  assumptions  about  usage  charges,  costs  for  the  two  approaches  were 
projected  thorugh  1989. 


RESULTS 

Instructional  Effectiveness 

The  mean  measures  obtained  for  the  classroom  and  PREST  groups  are  presented  in 
Table  1.  As  shown,  the  mean  entering  RGL,  which  was  based  on  the  confirmatory  Gates- 
MacGinitie  test,  is  one-half  grade  higher  for  the  total  PREST  group  than  for  the  total 
classroom  group.  As  indicated  previously,  this  difference  is  attributable  to  the  fact  that 
21  PREST  students  entered  the  ART  program  with  an  entering  RGL  between  6.0  and  6.7. 
Within  the  restricted  sample,  the  entering  RGLs  for  the  classroom  and  PREST  groups 
were  comparable.  Although  the  PREST  restricted  group  had  higher  AFQT  and  WK  scores 
than  did  the  classroom  group,  the  differences  were  not  significant. 

An  ANOVA  performed  on  scores  obtained  by  the  restricted  samples  on  the  NRRT  pre- 
and  posttests  showed  that  both  groups  improved  significantly  in  their  general  reading 
comprehensive  skills  during  the  instructional  treatment  (F(l,74)  =  82.0,  p  <  .001). 
However,  there  was  no  overall  significant  difference  between  the  groups  (F(l,74)  =  2.96,  p 
>  .05)  and,  more  importantly,  no  significant  difference  in  pre-  to  post-gain  between  groups 
(F(l,74)  =  0.5).  These  findings  were  verified  by  results  of  a  duplicate  ANOVA  performed 
on  the  Gates-MacGinitie  test  scores  obtained  by  the  two  groups.  Confirmation  came  from 
an  analysis  of  covariance  (ANCOVA)  performed  on  the  post-NRRT  scores  using  the  pre- 
NRRT  scores  as  a  covariate.  With  pretest  differences  removed,  there  was  no  statistically 


significant  difference  between  groups  on  the  post-NRRT  scores  (F(i,72)  =  1.18,  p  >  .25). 2 
Also,  the  difference  in  instructional  days  required  by  the  restricted  groups  was  not 
significant  (t  =  .82,  p  >  .2). 


Table  1 

Measures  Obtained  for  Restricted 
and  Overall  Samples 


Restricted  Sample8  Overall  Sample 

Measure  Classroom  PREST  Classroom  PREST 

(Mean)  (N=47)  (N=47)  (N=75)  (N=77) 


Background  Data 

Entering  RGL 

5.5 

5.6 

5.2 

5.7 

AFQT  score 

38.2 

40.6 

38.8 

40.3 

WK  score 

43.9 

44.3 

44.4 

44.9 

High  School  Graduates  (%) 

Immediate  Effects 

96.0 

96.0 

93.0 

92.0 

NRRT  score 

Pretest 

17.0 

17.6 

16.0 

17.7 

Posttest 

20.1 

21.1 

19.3 

21.0 

Gates-MacGinitie  test  scoreb 

Pretest 

39.7 

40.1 

-- 

-- 

Posttest 

42.2 

43.3 

— 

— 

Number  instructional  days 

11.3 

12.2 

13.1 

12.6 

Prolonged  Effects 

Recruit  Academic  Test  1  (%  correct) 

66 

66 

63 

68 

Recruit  Academic  Test  2  (%  correct) 

63 

67 

63 

66 

Recruit  Academic  Test  3  (%  correct) 

63 

65 

63 

66 

Recruit  Academic  Test  4  (%  correct) 

71 

71 

71 

71 

includes  only  those  students  with  RGLs  between  5.0  and  6.0. 

^Scores  not  included  for  the  overall  samples  because  of  the  ceiling  effect  noted. 


2The  NRRT  data  were  analyzed  by  two  methods,  both  yielding  the  same  conclusion. 
They  were:  (1)  a  2  x  2  x  2  repeated  measures  ANOVA  with  groups  and  the  blocking-factor 
test  forms  as  between  effects  and  the  tests  as  repeated  measures,  and  (2)  a  groups  by  test 
forms  ANCOVA  on  the  posttest  scores  using  the  estimated  true  pretest  score  as  the 
covariate.  Each  of  these  designs  has  its  advantages  for  the  analysis  of  intact  nonrandom 
groups  (Lord,  1967;  Kenney,  1975;  Cronback  &  Furby,  1970).  The  estimated  true  pretest 
scores  were  used  to  reduce  bias  in  the  ANCOVA  caused  by  error  in  obtained  pretest  scores 
(Reichardt,  1979;  Lord,  1956).  The  estimated  true  scores  were  calculated  by  multiplying 
deviation  scores  by  the  test-form  reliability  (Kuder-Richardson  20),  and  then  converting 
back  to  raw  scores. 


The  ANOVA  performed  on  recruit  academic  test  scores  obtained  by  restricted  sample 
members  completing  all  four  tests  revealed  no  overall  significant  differences  between 
groups  (F(l,64)  =  1.79,  p  >  .15)  and  no  significant  differential  change  between  groups  over 
tests  (F(3,i92)  =  1.74,  p  >  .15).  Both  groups  improved  significantly  from  Test  1  to  Test  4 
(F(3.192)  =  13.99,  p  <  .001),  raising  their  mean  score  from  66  to  71.  Thus,  the  prolonged 
benefits  provided  by  the  two  approaches  to  ART  were  equal,  with  the  two  groups  having 
equal  scores  and  gaining  by  the  same  amount. 

Finally,  the  attrition  of  the  groups  did  not  differ  significantly,  with  17  and  16  from 
the  classroom  and  PREST  groups  respectively  attriting  before  completion  of  recruit 
training.  Most  of  the  attrition  was  attributable  to  nonacademic  reasons;  only  three 
recruits  (one  from  the  classroom  group  and  two  from  the  PREST  group)  were  discharged 
because  of  academic  failure  in  recruit  training. 

Since  the  two  approaches  were  equally  effective,  it  was  not  necessary  to  quantify  any 
differences  in  instructional  advantage  in  terms  of  cost  savings.  However,  there  are  other 
aspects  of  PREST  worth  reviewing.  Observations  of  the  students  using  the  PLATO  system 
revealed  that  they  were  very  attentive  and  very  interested  in  their  training  and  that  one 
instructor  could  adequately  handle  24  students.  (The  two  military  personnel  at  Orlando 
who  served  as  the  PREST  instructors  stated  that  30  students  would  be  a  realistic 
maximum.)  Most  of  the  instructor  time  was  spent  in  answering  questions,  providing 
feedback,  or  moving  a  student  from  on-line  instruction  to  off-line  materials.  Although 
the  efficacy  of  the  off-line  materials  cannot  be  statistically  assessed  from  available  data, 
the  work-sheets  developed  by  RBS  appeared  to  be  quite  good  and  Navy  relevant. 

Cost  Effectiveness 


The  costs  for  one  year,  1980,  are  summarized  below: 


Classroom 


Instructional  staff 
Materials 

Projected  costs  for  1980 
PREST 


Terminals 

Usage  Charge 

Maintenance 

Instructors 

Subtotal 

Cost  avoidance 


$  72,940  (5  @  $14,588  each) 

$  2,188 

$  75,128 


$  23,520  (24  @  $4,900  each;  allocated 

over  5  years) 

$  76,320  ($265/terminal/month) 

$  10,368  ($36/terminal/month) 

$  29,176  (2  @  $14,588  each) 

$139,384 

$  7,022 


Projected  costs  for  1980  $132,362 


Figures  1  and  2  chart  the  time-course  behavior  of  the  projected  costs,  in  1980  dollars, 
for  military  and  civilian  instructors  respectively,  assuming  either  a  constant  cost  for 


1980  HOLLARS  IN  THOUSANDS  1980  DOLLARS  IN  THOUSANDS 


-v?\~ 


mm 


Figure  1.  Cost  comparisons  of  program  delivery  based  on  military 
instructors— 10%  per  year  decline  in  hardware  costs. 


PREST,  NO  CHANCE  IN 
USAGE  CHARGE 


CLASSROOM  APPROACH 

PREST,  Vi  PER  YEAR  DECLINE  IN 
USAGE  CHARGE 


Figure  2.  Cost  comparisons  of  program  delivery  based  on  civilian 
instructors— 10%  per  year  decline  in  hardware  costs. 
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usage  charge  or  a  5  percent  decline  in  this  charge.  With  the  former  assumption,  the  costs 
of  the  alternatives,  whether  based  on  using  military  or  civilian  instructors,  do  not  reach  a 
break-even  point  during  the  1980s.  With  the  latter,  the  costs  based  on  using  military 
instructors  would  break  even  about  1990;  and  the  costs  based  on  using  civilian  instructors, 
in  1987. 


DISCUSSION  AND  CONCLUSIONS 

Determining  whether  a  proposed  program  is  cost-effective  hinges  on  the  costs  of 
alternative  programs.  In  this  study,  the  classroom  approach  to  ART  was  found  to  be  more 
cost-effective  than  PREST  because  of  the  relatively  low  cost  of  military  instructors. 
Higher  instructor  costs  would  have  led  to  different  conclusions.  For  example,  in  a  recent 
Army  study  on  the  cost-effectiveness  of  PLATO  compared  to  classroom  basic  skills 
instruction,  it  was  concluded  that  PLATO  was  a  cost-effective  alternative  to  traditional 
instruction  (Blevins  &  Phipps,  1980).  However,  it  must  be  noted  that  this  study  did  not 
include  costs  for  the  instructor  managing  the  PLATO  classroom.  These  costs,  which 
covered  a  total  of  9408  instructional  hours  (56  Army  enlistees  and  168  hours  per  student) 
were  included  in  a  $44,000  contract  to  a  public  school  system  to  teach  basic  skills  to 
Army  enlistees.  This  amount  of  instruction  would  have  cost  about  $12,000  at  ART 
Orlando  if  military  instructors  were  used.  Although  the  Army  study  considered  the 
PLATO  system  to  be  instructor  independent,  this  effort  considered  that  PREST  required 
one  instructor  per  24  students.  Finally,  it  should  be  noted  that,  if  the  cost-comparison 
had  been  conducted  in  San  Diego,  where  salaries  for  civilian  instructors  are  higher  than  in 
Orlando,  the  break-even  point  for  the  PREST  and  classroom  approaches  with  civilian 
instructors  would  have  occurred  in  about  1984.  These  examples  illustrate  some  problems 
with  conclusions  from  cost-effective  studies  (c.f.,  Kazanowski,  1968). 

The  usage  charge  for  PLATO  with  its  centralized  mainframe  is  the  main  reason  why 
PREST  costs  more  than  does  the  classroom  approach  to  ART.  Although  the  mainframe 
will  continue  to  be  used  in  cases  where  instruction  must  be  managed  for  a  large  number  of 
students  or  a  large  mix  of  courses  is  offered  for  a  limited  time,  advances  in  computer 
technology  are  rapidly  making  the  stand-alone  systems  more  economically  attractive  than 
the  centralized  mainframe  (Micheli,  Morris,  &  Swope,  1980).  In  the  ART  environment, 
where  the  instruction  is  fixed  throughout  the  year,  it  does  not  seem  advantageous  to  pay  a 
subscription  fee  continually  for  a  limited  amount  of  courseware  and  to  send  the  same 
courseware  repeatedly  over  phone  lines.  This  problem  will  be  partly  remedied  by  the 
introduction  of  a  PLATO  system  that  uses  microprocessors  and  floppy  discs.  This 
configuration  still  requires  communication  to  the  mainframe  for  test  and  management 
functions,  possibly  over  a  reserved  terminal.  Furthermore,  the  durability  of  the  floppy 
discs  containing  the  basic  skills  courseware  must  be  determined  before  a  cost  analysis 
could  be  performed.  This  new  configuration,  however,  will  undoubtedly  improve  the  cost- 
effectiveness  of  computer-based  instruction. 

Overall  costs  could  also  be  decreased  if  the  instructor  staff  could  be  decreased 
without  a  corresponding  decrease  in  program  effectiveness.  In  the  PREST  approach  to 
ART,  instructors  route  students  to  off-line  materials  and  provide  feedback  regarding 
these  materials.  These  off-line  instructor  tasks  increase  instructor  dependence  that,  in 
turn,  increases  the  cost  of  the  overall  configuration.  Thus,  if  CAI  were  employed  only  in 
those  ART  modules  in  which  the  computer  is  instructor  independent,  savings  due  to 
decreased  instructor  costs  could  be  realized.  The  extent  to  which  this  instruction  could 
be  delivered  on  low-cost  microcomputers  would  lead  to  further  cost  benefits.  Wisher 
(1980  a,  b)  tested  an  instructor-independent  CAI  approach  to  the  phonics  and  vocabulary 
modules  of  ART.  Results  showed  that  the  CAI  phonics  instruction  was  as  efficient  as  the 
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classroom  instruction,  and  that  the  CAI  vocabulary  instruction  was  three  times  more 
efficient  than  the  classroom  instruction.  Thus,  it  appears  that  if,  for  example,  the  highly 
efficient  vocabulary  instruction  were  delivered  on  a  low-cost  device,  the  system  would  be 
highly  cost-effective  today.  This  assumes  that  one  instructor  is  eliminated,  and  the  costs 
of  the  hardware  are  allocated  over  5  years.  The  technology  is  now  available  to  deliver 
cost-effective  training  for  some  of  the  ART  modules. 

This  evaluation  of  PREST  did  not  test  a  nonnative  English  speaking  population.  Thus, 
it  is  not  known  how  effective  PREST  would  be  for  them.  Since  this  population  is  an 
increasing  concern  in  the  Navy  (Salas,  Kincaid,  &  Ashcroft,  1980),  plans  are  being  made  to 
develop  and  implement — in  FY  81 — a  separate  ART  module  for  them.  Any  computer- 
based  system  being  considered  as  a  substitute  for  classroom  instruction  must  be  suitable 
for  this  population. 

Because  of  the  varying  month-by-  month  throughput,  savings  could  probably  be 
realized  by  turning  off  some  terminals  during  the  winter  months.  The  increase  in  ART 
entrants  during  the  summer  months  would  require  that  instruction  be  staggered  through¬ 
out  the  day  or  that  the  off-line  materials  receive  more  usage.  In  the  classroom,  the 
student/instructor  ratio  fluctuates  during  the  year.  The  cost  for  the  varying  throughput, 
however,  affects  the  classroom  and  PREST  configurations  equally. 

The  off-line  worksheets  would  be  an  excellent  addition  to  the  ART  study  skills 
module.  Since  they  are  Navy  relevant,  they  would  complement  the  current  civilian¬ 
relevant  classroom  materials. 


RECOMMENDATIONS 

1.  The  PREST  program  should  not  be  substituted  for  the  current  ART  classroom 
approach  at  this  time. 

2.  Consideration  should  be  given  to  (a)  replacing  the  vocabulary  and  comprehension 
modules  of  ART  with  a  computer-based  system  and  (b)  using  low-cost,  stand-alone 
microcomputers  as  the  delivery  devices. 

3.  The  contractor-developed  worksheets  should  be  incorporated  into  the  ART  study 
skills  module. 

4.  The  NRRT  should  be  developed  further  as  a  selection  device  for  ART. 
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